1. Do existing pretrained models correctly recognize faces in images?¶
We will use 2 pretrained models with vgg16 and resNet50 architectures that are trained on the VGGface2 data set. VGGFace2 dataset is developed by researchers at the Visual Geometry Group at Oxford, includes faces across different poses and ages. The dataset contains 3.31 million images of 9131 subjects (including actors, athletes, politicians), approximately 362 images for each subject. This two pretrained models (VGG16 and ResNnet50 trained on VGGFACE2 dataset) are available in package 'keras_vggface' available (https://github.com/rcmalli/keras-vggface). this data set has a 8631 to 500 train and validation split on the identities.
Input for this two models require a face detector. For that reason, I use the detector MTCNN in package mtcnn.mtcnn. The 'preprocess_input' function of the 'keras_vggface.utils' package is used to prepare the face to become the input of the models.(required size is 224 × 224 crop of the center face image is used as input to the network). The face descriptor or the embeddings is extracted from the layer adjacent to the classifier layer. This leads to a 2048 dimensional descriptor, which is then L2 normalized.
When specifying weights: one of None (random initialization) or "vggface" (pre-training on VGGFACE datasets). this work project only uses weight pretrained on 'vggface'. This project uses the following dataset for prediction:
Selected 11 classes of celebrities in CASIA-WebFace publicly available dataset(here) This dataset is created by the Institute of Automation, Chinese Academy of Sciences (CASIA) gathered from IMDb profiles. Labels and image list of CASIA WebFace data set can be found here. CASIA WebFace Facial dataset includes 494,414 images over 10,575 identities and requires some filtering for quality. The number of images in each class averages 46 image per subject, however, for this project 11 classes with at least 180 images are selected.
- candidate images for prediction are in 'test_dir'
In [1]:
! pip show keras-vggface
In [2]:
import keras_vggface
print(keras_vggface.__version__)
In [3]:
! python --version
In [5]:
import os
# os.environ['KMP_DUPLICATE_LIB_OK']='True'from __future__ import absolute_import, division, print_function, unicode_literals
import pathlib
import os
import matplotlib.pyplot as plt
from matplotlib import pyplot
import numpy as np
from numpy import expand_dims
from numpy import asarray
import pandas as pd
import seaborn as sns
from IPython import display
from PIL import Image
from IPython import display
import cv2
import keras_vggface
import keras
from keras import layers
from keras import models
from keras import regularizers
from keras import optimizers
from keras.preprocessing import image
from keras.preprocessing.image import ImageDataGenerator, array_to_img, img_to_array, load_img
from keras.models import load_model
from keras.layers import *
from keras.models import Sequential
from keras_vggface.vggface import VGGFace
from keras_vggface.utils import preprocess_input
from keras_vggface.utils import decode_predictions
from keras import regularizers
from keras.utils import to_categorical
from mtcnn.mtcnn import MTCNN
In [10]:
IMG_SIZE = 224
IMG_SHAPE = (IMG_SIZE, IMG_SIZE, 3)
required_size = (IMG_SIZE, IMG_SIZE)
# Tesing set
test_dir = 'keras-facenet/data/test'
listGroupsTest = os.listdir(test_dir)
listGroupsTest = [f for f in listGroupsTest if not f.startswith('.')]
listGroupsTest
Out[10]:
In [114]:
required_size = (IMG_SIZE, IMG_SIZE)
In [99]:
def extract_face(img_path, required_size=required_size):
# load image from file
img = cv2.imread(img_path)
img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
# create detector, using default weights
detector = MTCNN()
# detect faces in the image
results = detector.detect_faces(img_rgb)
# extract the bounding box associated to larger area
bigger_face = max(results, key=lambda b: b['box'][2] * b['box'][3])
x1, y1, width, height = bigger_face['box']
x2, y2 = x1 + width, y1 + height
# extract the face
face = img_rgb[y1:y2, x1:x2]
image = Image.fromarray(face)
image = image.resize(required_size)
face_array = asarray(image)
return face_array
In [71]:
# example of face extraction:
img_path = 'Data/train/Colin_Firth/001.jpg'
img = cv2.imread(img_path)
img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
plt.imshow(img_rgb)
plt.show()
face = extract_face(img_path)
plt.imshow(face)
plt.show()
Predict using pretrained models:¶
In [11]:
def prepare_image_pretrained(img_path, img_size = IMG_SIZE):
face = extract_face(img_path)
face = face.astype('float32')
face = expand_dims(face, axis=0)
samples = keras_vggface.utils.preprocess_input(face, version=2)
return samples
def faceClassifier_pretrained(model, imagePath):
preprocessed_image = prepare_image_pretrained(imagePath)
# Use pretrained model to classify the image
predictions = model.predict(preprocessed_image)
results = decode_predictions(predictions)
return results
In [12]:
def faceRecognition_pretrained(model, test_dir):
predictions_list=[]
for celeb in os.listdir(test_dir):
if not celeb.startswith('.'):
test_img_path = os.path.join(test_dir, celeb)
if not test_img_path.startswith('.'):
for test_img in os.listdir(test_img_path):
if not test_img.startswith('.'):
fullimagepath = os.path.join(test_img_path, test_img)
face = prepare_image_pretrained(fullimagepath)
predictions = model.predict(face)
predictions_list.append(predictions)
predictions = np.sum(predictions_list, axis=0)
top_pred = decode_predictions(predictions)
print('Test picture from: ', celeb,':\n Prediction: ' ,top_pred, '\n')
VGG16¶
In [13]:
# Lets predict using exisiting pretrained vgg16 model
vgg16 = VGGFace(model='vgg16')
faceRecognition_pretrained(vgg16, test_dir)
In [16]:
vgg16.summary()
ResNet50¶
In [14]:
# Lets predict using exisiting pretrained resnet50 model
resnet50 = VGGFace(model='resnet50')
faceRecognition_pretrained(resnet50, test_dir)
In [15]:
resnet50.summary()
Report:¶
Pretrained models performance:
Although pretrained models include the classes asssociated to selected subjects, they do not do a good job in predicting them.
VGG15 predicts only 3 our of the 11 pictures in the tes_dir, where as the Resnet50 None of them.
2. How to improve performance of existing pretrained models using deep learning techniques?¶
Some explanation on the namings:
-conv_base is used to expleain that it respresents the convolutional base and does not have the top layer
-FE used to specify the object is used in the Feature Extraction
-TL used to specify the object is used in the Transfer Learning
-FT used to specify the object is used in the Fine Tuning Feature Extract (FE):¶
Extract features and labels in train, validation, and test folders by predicting from the convolutional base of the pretrained models Created Models using Extracted Features from Convolutional Base of VGG16 and ResNet50 Reshape them to be inputted into a model with dense layers Use dropouts and regularizes for regularization to avoid overfitting.
In [101]:
# create a plot function to avoid rewriting:
In [17]:
def plot_history(history, title, clr1='blue', clr2='green'):
# helper function to plot the loss and accuracy
plt.figure(figsize=(18, 7))
acc = history.history['accuracy']
val_acc = history.history['val_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']
plt.suptitle('Categorical Plotting')
plt.subplot(1, 2, 1)
plt.plot(range(1, len(acc)+1), acc, color=clr1, label='Training Accuracy ' )
plt.plot(range(1, len(acc)+1), val_acc, color=clr2, linestyle='dashed', label='Validation Accuracy' )
plt.legend()
plt.ylabel('Accuracy')
plt.ylim([min(plt.ylim()),1])
plt.title('Training and Validation Accuracy')
plt.subplot(1, 2, 2)
plt.plot(range(1, len(acc)+1), loss, color=clr1, label='Training Loss ')
plt.plot(range(1, len(acc)+1), val_loss, color=clr2, linestyle='dashed', label='Validation Loss')
plt.legend()
plt.ylabel('Loss')
plt.title('Training and Validation Loss')
plt.xlabel('epoch')
plt.suptitle(title)
plt.show()
Loading the convolutional base of model VGG16 for FE:¶
In [108]:
conv_base_vgg16_FE = VGGFace(input_shape=IMG_SHAPE,model='vgg16',weights='vggface',include_top=False)
In [110]:
conv_base_vgg16_FE.summary()
Extract features and labels: for train validation and test, and reshape them to get prepared for the required input dimension for a model with dense layers: (input_dim=7 7 512)
In [104]:
base_dir = 'Data'
test_dir = 'keras-facenet/data/test'
train_dir = os.path.join(base_dir, 'train')
validation_dir = os.path.join(base_dir, 'val')
datagen = ImageDataGenerator(rescale=1./255)
batch_size = 20
def extract_features_vgg16(directory, sample_count):
features = np.zeros(shape=(sample_count, 7, 7, 512))
labels = np.zeros(shape=(sample_count))
generator = datagen.flow_from_directory(
directory,
target_size=(IMG_SIZE, IMG_SIZE),
batch_size=batch_size,
class_mode='sparse')
i = 0
for inputs_batch, labels_batch in generator:
features_batch = conv_base_vgg16_FE.predict(inputs_batch)
features[i * batch_size : (i + 1) * batch_size] = features_batch
labels[i * batch_size : (i + 1) * batch_size] = labels_batch
i += 1
if i * batch_size >= sample_count:
# Note that since generators yield data indefinitely in a loop,
# we must `break` after every image has been seen once.
break
return features, labels
# extract the features
train_features_vgg16, train_labels_vgg16 = extract_features_vgg16(train_dir, 2142)
validation_features_vgg16, validation_labels_vgg16 = extract_features_vgg16(validation_dir, 1089)
test_features_vgg16, test_labels_vgg16 = extract_features_vgg16(test_dir, 11)
train_features_vgg16 = np.reshape(train_features_vgg16, (2142, 7*7* 512))
validation_features_vgg16= np.reshape(validation_features_vgg16, (1089, 7* 7* 512))
test_features_vgg16 = np.reshape(test_features_vgg16, (11, 7* 7* 512))
In [20]:
vgg16_FE = keras.models.Sequential()
vgg16_FE.add(keras.layers.Dense(256, activation='relu', input_dim=7 * 7 * 512))
vgg16_FE.add(keras.layers.Dropout(0.5))
vgg16_FE.add(keras.layers.Dense(128, activation='relu'))
vgg16_FE.add(keras.layers.Dropout(0.5))
vgg16_FE.add(keras.layers.Dense(11, activation='softmax'))
#compile model
vgg16_FE.compile(optimizer=keras.optimizers.Adam(lr=2e-2),
loss='sparse_categorical_crossentropy',metrics=['accuracy'])
# fit model using 20 epochs
history_vgg16_FE= vgg16_FE.fit(train_features_vgg16, train_labels_vgg16,
epochs=20,
batch_size=20,
validation_data=(validation_features_vgg16, validation_labels_vgg16))
In [21]:
vgg16_FE.save('vgg16_FE.h5')
In [22]:
plot_history(history_vgg16_FE, 'Feature Extract - created model from extracted features of vgg16 convolutional base')
Loading the convolutional base of model resnet50 for FE¶
In [24]:
conv_base_resnet50_FE = VGGFace(input_shape=IMG_SHAPE,model='resnet50',weights='vggface',include_top=False)
In [105]:
base_dir = 'Data'
test_dir = 'keras-facenet/data/test'
train_dir = os.path.join(base_dir, 'train')
validation_dir = os.path.join(base_dir, 'val')
datagen = ImageDataGenerator(rescale=1./255)
batch_size = 20
def extract_features_resnet50(directory, sample_count):
features = np.zeros(shape=(sample_count, 1, 1, 2048))
labels = np.zeros(shape=(sample_count))
# Data Augmentation
generator = datagen.flow_from_directory(
directory,
target_size=(IMG_SIZE, IMG_SIZE),
batch_size=batch_size,
class_mode='sparse')
i = 0
for inputs_batch, labels_batch in generator:
features_batch = conv_base_resnet50_FE.predict(inputs_batch)
features[i * batch_size : (i + 1) * batch_size] = features_batch
labels[i * batch_size : (i + 1) * batch_size] = labels_batch
i += 1
# print(inputs_batch, labels_batch )
break
if i * batch_size >= sample_count:
# Note that since generators yield data indefinitely in a loop,
# we must `break` after every image has been seen once.
break
return features, labels
train_features_resnet50, train_labels_resnet50 = extract_features_resnet50(train_dir, 2142)
validation_features_resnet50, validation_labels_resnet50 = extract_features_resnet50(validation_dir, 1089)
test_features_resnet50, test_labels_resnet50 = extract_features_resnet50(test_dir, 11)
In [28]:
train_features_resnet50 = np.reshape(train_features_resnet50, (2142, 1 * 1 * 2048))
validation_features_resnet50 = np.reshape(validation_features_resnet50, (1089, 1 * 1 * 2048))
test_features_resnet50 = np.reshape(test_features_resnet50, (11, 1 * 1 * 2048))
In [29]:
resnet50_FE = keras.models.Sequential()
resnet50_FE.add(keras.layers.Dense(512, kernel_regularizer=regularizers.l2(lamb), activation='relu', input_dim=1 * 1 * 2048))
resnet50_FE.add(keras.layers.Dropout(0.5))
resnet50_FE.add(keras.layers.Dense(128, kernel_regularizer=regularizers.l2(lamb), activation='relu'))
resnet50_FE.add(keras.layers.Dropout(0.5))
resnet50_FE.add(keras.layers.Dense(11, activation='softmax'))
resnet50_FE.compile(optimizer=keras.optimizers.Adam(lr=2e-2),
loss='sparse_categorical_crossentropy',metrics=['accuracy'])
history_resnet50_FE = resnet50_FE.fit(train_features_resnet50, train_labels_resnet50,
epochs=20,
batch_size=20,
validation_data=(validation_features_resnet50, validation_labels_resnet50))
In [30]:
resnet50_FE.summary()
In [31]:
plot_history(history_resnet50_FE, 'Feature Extract - created model from extracted features of resnet50 ')
In [33]:
resnet50_FE.save('resnet50_FE_30epochs.h5')
Report:¶
The accuracy of training and test remains almost constant through different epochs. Also the loss of training and validation becomes almost equal in 20 epochs. No need for training with more epochs.
The validation accuracy remains less that training, but they are close.
20 epochs seems to be a good choice, accuracy has been constant and loss values are decreasing across both training and validation.
Transfer Learning (TL)¶
Build a model on top of the convolutional base of the VGG16 and ResNet50 Freeze the parameters of the convolutional base Train parameters in added dense and classification layers Use dropouts and regularizes for regularization to avoid overfitting
Data Augmentation¶
- Images are augmented based on horizontal flip, 45 rotation, and zooming for training. The validation images are not augmented.
In [37]:
pathFolder = './Data/'
IMG_SIZE = 224
IMG_SHAPE = (IMG_SIZE, IMG_SIZE, 3)
# image augmentation
# We are getting our train_generator, validation_generator and test_generator ready :
train_datagen=ImageDataGenerator(preprocessing_function=keras_vggface.utils.preprocess_input,
horizontal_flip=True,
rotation_range=45,
zoom_range=[0.8,1.0])
test_datagen=ImageDataGenerator(preprocessing_function=keras_vggface.utils.preprocess_input)
# Training set
pathTrain = pathFolder + 'train/'
listGroupsTrain = os.listdir(pathTrain) # the directory path
listGroupsTrain = [f for f in listGroupsTrain if not f.startswith('.')]
# Validation set
pathVal = pathFolder + 'val/'
listGroupsValid = os.listdir(pathVal) # the directory path
listGroupsValid = [f for f in listGroupsValid if not f.startswith('.')]
# Tesing set
test_dir = 'keras-facenet/data/test'
listGroupsTest = os.listdir(test_dir)
listGroupsTest = [f for f in listGroupsTest if not f.startswith('.')]
# Load the data into the ImageDataGenerator
train_generator=train_datagen.flow_from_directory(pathFolder+'train',
target_size=(224,224),
color_mode='rgb',
batch_size=64,
class_mode='categorical',
shuffle=True,
classes=listGroupsTrain)
validation_generator=test_datagen.flow_from_directory(pathFolder+'val',
target_size=(224,224),
color_mode='rgb',
batch_size=64,
class_mode='categorical',
shuffle=False,
classes=listGroupsValid)
test_generator = test_datagen.flow_from_directory(test_dir,
target_size=(224,224),
color_mode='rgb',
batch_size=64,
class_mode='categorical')
Steps Transfer learning :¶
- Get the convolutional base from the pretrained model
- Freeze the layers in the convolutional base
- Add dense layers and the classifiction layer to adjust added layer's weight by retrain model to my dataset's categories
- In one model, 1 hidden layer is added, in the next model, 2 hidden layers are added along with the classifcation layer.Transfer Learning on ResNet50 :¶
Added 1 dense and a softmax layer and retrained the added layers:¶
- Having conv base layers frozen.
In [38]:
conv_base_resnet50 = VGGFace(input_shape=IMG_SHAPE,model='resnet50',
weights='vggface',include_top=False)
conv_base_resnet50.trainable = False
# Regularize the dense layer by using L1
kernel_weight = 0.02
bias_weight = 0.02
hidden_dim =512
resnet50_TL_1layer = Sequential([
conv_base_resnet50,
keras.layers.Flatten(),
keras.layers.Dense(hidden_dim, activation='relu', name="FC_1",
kernel_regularizer=regularizers.l1(kernel_weight),
bias_regularizer=regularizers.l1(bias_weight)),
keras.layers.Dense(11, activation='softmax', name='FC_2',
kernel_regularizer=regularizers.l1(kernel_weight),
bias_regularizer=regularizers.l1(bias_weight))])
# compile
resnet50_TL_1layer.compile(optimizer=keras.optimizers.RMSprop(lr=1e-5),
loss='categorical_crossentropy' ,metrics=['accuracy'])
# fit the model using similar hyperparameters as before
step_size_train=train_generator.n//train_generator.batch_size
history_resnet50_TL_1layer= resnet50_TL_1layer.fit_generator(generator=train_generator,
validation_data=validation_generator,
steps_per_epoch=step_size_train,
epochs=20)
resnet50_TL_1layer.save('resnet50_TL_1layer.h5')
resnet50_TL_1layer.save_weights('resnet50_TL_1layer_weights.h5')
plot_history(history_resnet50_TL_1layer, 'Transfer learning - Resnet50 - re-trained added 1 hidden dense and output layers')
In [126]:
resnet50_TL_1layer.save('resnet50_TL_1layer.h5')
In [111]:
plot_history(history_resnet50_TL_1layer, 'Transfer learning - Resnet50 - re-trained added 1 hidden dense and output layers')
In [125]:
FaceRecognition_testdata(resnet50_TL_1layer, listGroupsTest)
Added 2 dense and a softmax layer and retrained the added layers:¶
- Having conv base layers frozen.
In [41]:
conv_base_resnet50 = VGGFace(input_shape=IMG_SHAPE,model='resnet50',
weights='vggface',include_top=False)
In [44]:
conv_base_resnet50.trainable = False
number_groupsTrain = len(listGroupsTrain)
# Regularize the dense layer by using L1
kernel_weight = 0.02
bias_weight = 0.02
hidden_dim =512
resnet50_TL = Sequential([
conv_base_resnet50, # base from resnet50
keras.layers.Flatten(),
keras.layers.Dense(hidden_dim, activation='relu', name="FC_1"),
keras.layers.Dense(hidden_dim, activation='relu', name="FC_2",
kernel_regularizer=regularizers.l1(kernel_weight),
bias_regularizer=regularizers.l1(bias_weight)),
keras.layers.Dense(11, activation='softmax', name='FC_3',
kernel_regularizer=regularizers.l1(kernel_weight),
bias_regularizer=regularizers.l1(bias_weight))])
resnet50_TL.summary()
In [45]:
resnet50_TL.compile(optimizer=keras.optimizers.RMSprop(lr=1e-5),
loss='categorical_crossentropy' ,metrics=['accuracy'])
step_size_train=train_generator.n//train_generator.batch_size
history_resnet50_TL = resnet50_TL.fit_generator(generator=train_generator,
validation_data=validation_generator,
steps_per_epoch=step_size_train,
epochs=20)
In [123]:
resnet50_TL.save('resnet50_TL.h5')
resnet50_TL.save_weights('resnet50_TL_weights.h5')
In [103]:
plot_history(history_resnet50_TL, 'Transfer learning - resnet50 - re-trained added 2 hidden dense and output layers')
In [ ]:
# Create a function that goes through all images in test_dir and predicts them using the specified model:
In [52]:
def FaceRecognition_testdata(model, listGroupsTest):
i=0
for celeb in listGroupsTest:
if not celeb.startswith('.'):
test_img_path = os.path.join(test_dir, celeb)
if not test_img_path.startswith('.'):
for test_img in os.listdir(test_img_path):
if not test_img.startswith('.'):
test_identity = 'Not in database'
test_img_fullpath = os.path.join(test_img_path, test_img)
face = extract_face(test_img_fullpath)
face = face.astype('float32')
face = expand_dims(face, axis=0)
preprocessed_face = preprocess_input(face, version=2)
#normalzed the color code
# preprocessed_face = face / 255
prediction = model.predict(preprocessed_face)
if np.max(prediction) > 0.5:
name = np.argmax(prediction, axis=-1)
test_identity = listGroupsTest[name[0]]
img = cv2.imread(os.path.join(test_img_path, test_img))
img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
plt.imshow(img_rgb)
plt.title(test_identity)
plt.show()
# print('test image is : ', test_identity)
In [53]:
FaceRecognition_testdata(resnet50_TL, listGroupsTest)
In [122]:
FaceRecognition_testdata(resnet50_TL, listGroupsTest)
Transfer Learning on VGG16 :¶
Add 1 hidden dense and output layer on the convolutional base and retrain added layers:¶
- Having conv base layers frozen.
In [106]:
conv_base_vgg = VGGFace(input_shape=IMG_SHAPE,model='vgg16',
weights='vggface',include_top=False)
In [60]:
conv_base_vgg.trainable = False
# Regularize the dense layer by using L1
kernel_weight = 0.02
bias_weight = 0.02
hidden_dim =512
vgg16_TL_1layer = Sequential([
conv_base_vgg,
keras.layers.Flatten(),
keras.layers.Dense(hidden_dim, activation='relu', name="FC_1",
kernel_regularizer=regularizers.l1(kernel_weight),
bias_regularizer=regularizers.l1(bias_weight)),
keras.layers.Dense(11, activation='softmax', name='FC_2',
kernel_regularizer=regularizers.l1(kernel_weight),
bias_regularizer=regularizers.l1(bias_weight))])
vgg16_TL_1layer.compile(optimizer=keras.optimizers.RMSprop(lr=1e-5),
loss='categorical_crossentropy' ,metrics=['accuracy'])
step_size_train=train_generator.n//train_generator.batch_size
history_vgg16_TL_1layer= vgg16_TL_1layer.fit_generator(generator=train_generator,
validation_data=validation_generator,
steps_per_epoch=step_size_train,
epochs=20)
vgg16_TL_1layer.save('vgg16_TL_1layer.h5')
vgg16_TL_1layer.save_weights('vgg16_TL_1layer_weights.h5')
plot_history(history_vgg16_TL_1layer, 'Transfer learning - vgg16 - re-trained added 1 hidden dense and output layers')
Add 2 hidden dense layers and output layer on the convolutional base and retrain added layers:¶
- having conv base layers frozen.
In [62]:
conv_base_vgg = VGGFace(input_shape=IMG_SHAPE,model='vgg16',weights='vggface',include_top=False)
conv_base_vgg.trainable = False
# Regularize the dense layer by using L1
kernel_weight = 0.02
bias_weight = 0.02
hidden_dim =512
vgg16_TL = Sequential([
conv_base_vgg,
keras.layers.Flatten(),
keras.layers.Dense(hidden_dim, activation='relu', name="FC_1",
kernel_regularizer=regularizers.l1(kernel_weight),
bias_regularizer=regularizers.l1(bias_weight)),
keras.layers.Dense(hidden_dim, activation='relu', name="FC_2",
kernel_regularizer=regularizers.l1(kernel_weight),
bias_regularizer=regularizers.l1(bias_weight)),
keras.layers.Dense(11, activation='softmax', name='FC_3',
kernel_regularizer=regularizers.l1(kernel_weight),
bias_regularizer=regularizers.l1(bias_weight))])
vgg16_TL.summary()
In [64]:
vgg16_TL.compile(optimizer=keras.optimizers.RMSprop(lr=1e-5),
loss='categorical_crossentropy' ,metrics=['accuracy'])
step_size_train=train_generator.n//train_generator.batch_size
history_vgg16_TL= vgg16_TL.fit_generator(generator=train_generator,
validation_data=validation_generator,
steps_per_epoch=step_size_train,
epochs=20)
In [65]:
vgg16_TL.save('vgg16_TL.h5')
vgg16_TL.save_weights('vgg16_TL_weights.h5')
In [66]:
# vgg16_TL = load_model('vgg16_TL.h5')
FaceRecognition_testdata(vgg16_TL, listGroupsTest)
In [67]:
plot_history(history_vgg16_TL, 'Transfer learning - vgg16 - re-trained added 2 hidden dense and output layers ')
Report:¶
Adding 2 hidden dense layers versus 1 hidden dense layer significantly improved the accuracy and the performance of the model created on Resnet50's convolutional base (val acc increased from approximately 76% to approximately 94%).
However, in the case of models using VGG16's convolutional base, the accuracy of the performance in both cases are ~ 94%
Validation and training accuracy and loss change together with a slight difference. The model is neither overfitting not underfitting
20 epochs seems to be a good choice since loss has been decreasing and train and validation accuracy are very close. Neither overfitting or underfitting.
Fine tunning¶
- Build a model on top of the convolutional base of the VGG16 and ResNet50
- Unfreeze the parameters of all layers in the last convolutional block
- The remaining layers in convolutional base are frozen
- Train parameters in the last convolutional block, added dense and classification layers
- Use regularizes for regularization to avoid overfitting
Fine Tuning on ResNet50:¶
In [72]:
conv_base_resnet_FT = VGGFace(input_shape=
IMG_SHAPE, model='resnet50',weights='vggface', include_top=False)
# The number of the layers in the base model
NumOfLayers = len(conv_base_resnet_FT.layers)
print("Number of layers in the base model: ", NumOfLayers)
In [74]:
conv_base_resnet_FT.trainable = True
for layer in conv_base_resnet_FT.layers[0:141]:
layer.trainable = False
i+=1
In [76]:
number_groupsTrain = len(listGroupsTrain)
# Regularize the dense layer by using L1
kernel_weight = 0.02
bias_weight = 0.02
hidden_dim =512
resnet50_FT = Sequential([
conv_base_resnet_FT,
keras.layers.Flatten(),
keras.layers.Dense(hidden_dim, activation='relu', name="FC_1",
kernel_regularizer=regularizers.l1(kernel_weight),
bias_regularizer=regularizers.l1(bias_weight)),
keras.layers.Dense(11, activation='softmax', name='FC_2',
kernel_regularizer=regularizers.l1(kernel_weight),
bias_regularizer=regularizers.l1(bias_weight))
])
resnet50_FT.summary()
In [77]:
resnet50_FT.compile(optimizer=keras.optimizers.RMSprop(lr=1e-5),
loss='categorical_crossentropy' ,metrics=['accuracy'])
step_size_train=train_generator.n//train_generator.batch_size
history_resnet50_FT= resnet50_FT.fit_generator(generator=train_generator,
validation_data=validation_generator,
steps_per_epoch=step_size_train,
epochs=20)
In [78]:
resnet50_FT.save('resnet50_FT.h5')
resnet50_FT.save_weights('resnet50_FT_weights.h5')
In [79]:
plot_history(history_resnet50_FT, 'Fine tuning - resnet50 - last convolutional block (conv5) and added dense layers re-trained')
In [80]:
FaceRecognition_testdata(resnet50_FT, listGroupsTest)
Fine Tuning on VGG16:¶
In [107]:
# Fine-tune only the last convolution
conv_base_vgg_FT = VGGFace(input_shape=
IMG_SHAPE, model='vgg16',weights='vggface', include_top=False)
In [88]:
# The number of the layers in the base model
NumOfLayers = len(conv_base_vgg_FT.layers)
print("Number of layers in the base model: ", NumOfLayers)
In [90]:
number_groupsTrain = len(listGroupsTrain)
# Regularize the dense layer by using L1
kernel_weight = 0.02
bias_weight = 0.02
hidden_dim =512
vgg16_FT = Sequential([
conv_base_vgg_FT,
keras.layers.Flatten(),
keras.layers.Dense(hidden_dim, activation='relu', name="FC_1",
kernel_regularizer=regularizers.l1(kernel_weight),
bias_regularizer=regularizers.l1(bias_weight)),
keras.layers.Dense(11, activation='softmax', name='FC_2',#)])
kernel_regularizer=regularizers.l1(kernel_weight),
bias_regularizer=regularizers.l1(bias_weight))])
vgg16_FT.summary()
In [91]:
conv_base_vgg_FT.trainable = True
for layer in conv_base_vgg_FT.layers[0:15]:
layer.trainable = False
i+=1
In [92]:
for layer in conv_base_vgg_FT.layers:
print(layer, layer.trainable)
In [93]:
vgg16_FT.compile(optimizer=keras.optimizers.RMSprop(lr=1e-5),
loss='categorical_crossentropy' ,metrics=['accuracy'])
step_size_train=train_generator.n//train_generator.batch_size
history_vgg16_FT= vgg16_FT.fit_generator(generator=train_generator,
validation_data=validation_generator,
steps_per_epoch=step_size_train,
epochs=20)
In [94]:
vgg16_FT.save('vgg16_FT.h5')
vgg16_FT.save_weights('vgg16_FT_weights.h5')
In [96]:
plot_history(history_vgg16_FT, 'Fine tuning - vgg16 - last convolutional block (conv5) and added Dense layers re-trained')
In [97]:
FaceRecognition_testdata(vgg16_FT, listGroupsTest)
Report:¶
Finetuning improved the performance accuracies more than transfer learning in models using both architectures
Model using VGG16 convolutional base responded better than ReNet50 to the transfer learning regardless of the 1 or 2 layers added ( with accuracy ~93%. val-acc ~94%). Model using ReNet50 convolutional base only compensated to the same accuracy when 2 hidden dense layers were added
Model using Resnet50 convolutional base responded to finetuning as both the training and validation accuracy increased to approximately 98%.
Highlight and Explanation of Reports:¶
- Repurposing existing models, using a moderate size dataset, to match an extracted face to a known individual can significantly improve the prediction accuracy of pretrained models.
- The performance of repurposed models strongly depends on the underlying network architecture and not necessarily the depth or number of layers. VGG16 yields higher validation accuracy with less layers retrained. VGG16 and Resnet50 models are both pretrained on the same VGGFace2 dataset. However, when more of the layers are retrained, model using ResNet50 improved more.
- Fine tuning improved the validation accuracies more than transfer learning in models created VGG16 and ResNet50’s convolutional base.
- Models using VGG16’s convolutional base responded better than ReNet50 to the transfer learning regardless of the 1 or 2 layers added. Models using ReNet50’s convolutional base only compensated to the same accuracy when 2 hidden dense layers were added.
- Model using Resnet50’s convolutional base responded better than VGG16 to finetuning. (model using VGG16 convolutional base improved from ~93% in TL to ~ 95% in FT, while using Resnet50 improved from ~93% in TL to ~97% in FT)
- Models in Transfer learning and finetuning provided a better fit than models created using extracted features. In transfer learning and fine tuning sections, training and validation accuracy improved together and remain very close. Models are neither overfitting nor underfitting. On the other hand, in the models created using extracted features, the validation accuracy remain less training accuracy.
- The performance of repurposed models strongly depends on the number of images in the underlying datasets of pretrained models, as FaceNet with 1 million identities strongly over performs VGG16 and ResNet50 with 9,131 identities.
- The initial dataset used to train them and not necessarily the depth or number of layers.